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ABSTRACT 


Successful landing of an autonomous unmanned aerial vehicle requires a high 
degree of accuracy and efficient, real-time processing. This research applies systems 
engineering concepts to investigate the feasibility of applying computer vision techniques 
and visual feedback in the control loop for an autonomous system. This thesis examines 
the framework and performance of an algorithm designed to detect and track a runway in 
images captured from a camera onboard an aircraft during the final approach and landing 
stages of flight. Using a series of image processing techniques to localize the runway and 
the Hough transformation for line detection, the algorithm is capable of detecting the 
edges of a runway with over 96 percent accuracy through 3000 test images. The 
operating conditions for this algorithm include any scenario in which visual flight rules 
apply. Additionally, the system will perform with runways that comply with Federal 
Aviation Administration regulations. Future applications of this algorithm should include 
aircraft attitude and pose estimation as well as full integration into an autonomous aircraft 
control system. 
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EXECUTIVE SUMMARY 


Research concerning autonomous vehicles and computer vision has expanded 
rapidly, and the two fields are closely related. The use of autonomous unmanned aerial 
vehicles (UAV) by the Department of Defense (DOD) has become a vital component of 
national security with the most recent successes demonstrated by the RQ-4 Global Hawk 
and the X-47B. Both of these aircraft are capable of autonomous takeoff, mission 
execution, and landing. These systems use an intricate differential global positioning 
system (DGPS) to collect accurate position estimates that allow for successful landings. 
This research follows a systems engineering approach to evaluating the feasibility of 
using computer vision and visual feedback techniques to control the aircraft in the final 
approach and landing stages of flight. Specifically, this evaluates framework for an 
algorithm to detect and track a runway from a camera onboard an aircraft. This thesis 
offers an improvement to current vision-based runway detection frameworks such as 
those made relevant by Shang and Zhongke (2011) in their paper titled, “Vision-Based 
Runway Recognition for Autonomous Landing of a UAV.” 

The first stages of image processing and detection use common pre-processing 
techniques, the first being thresholding. While thresholding is usually applied to the 
intensity of grayscale images, this research favors thresholding in the hue, saturation, and 
value (HSV) color space. This allows the algorithm to localize the position of the runway 
using chromaticity information as well as intensity information. The additional 
information improves the performance of the algorithm and results in improved isolation 
of the runway area and elimination of noise and clutter. The algorithm uses a hue 
threshold between 0 and 0.440 and between 0.700 and 1.0 to eliminate color ranges in the 
blue spectrum that easily coincide with the color of the sky. Additionally, the saturation 
threshold is set between 0 and 0.2 and the value threshold is set between 0.637 and 1.0 to 
capture the lighter and brighter runway markings. 

Following HSV thresholding, a series of morphological operations are applied to 
the image to eliminate clutter and noise while ensuring the entirety of the runway area is 

captured. The first of the basic operations used is dilation, which increases the perimeter 
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of a binary region in order to capture lost portions of the area within the region of interest 
(MathWorks 2017). The second basic operation is dilation, which decreases the perimeter 
of a binary region in order to eliminate clutter or unwanted areas (MathWorks 2017). 
Together, these operations form the basis of all morphological tasks. The algorithm first 
applies a filling operation, which dilates then erodes the image to fill holes in a region 
and recapture lost information. The algorithm then applies an opening operation to 
eliminate the noise and clutter in the image, specifically any grouping of pixels with a 
perimeter smaller than 30 pixels. Finally, the image is dilated with a vertical, rectangular 
structuring element to increase any portions of the runway region that may have been lost 
in the previous operations. Upon completion, the algorithm applies a second filling 
operation to ensure the remaining regions contain no unwanted gaps or holes. The result 
of these morphological operations is a masked image that localizes the runway area and 
eliminates as much noise and irrelevant area as possible. 

To reduce noise in the image resulting from internal camera operations or the 
environment, the algorithm filters the masked image to reduce the effects of Gaussian, 
salt and pepper, and quantization noise. The median filter was chosen instead of the mean 
filter or other techniques due to its robustness and edge preservation characteristics. As 
Dangeti (2003, 18) describes in her research, median filtering samples a predetermined 
window of pixels from the image and replaces the center pixel of interest with the median 
intensity value. 

In an additional attempt to preserve edges within each frame, the algorithm 
sharpens the image using MATLAB’s “imsharpen” function with an edge radius of one 
pixel. The algorithm uses a particular image sharpening technique, called unsharp 
masking, which creates a Gaussian-blurred version of the original image and subtracts the 
blurred copy from the original, resulting in sharper edges in areas that meet the 
conditions of the system (Cambridge in Colour 2017). This process advantageously 
produces edges that are sharply delineated and easier to detect in later stages of the 
algorithm. 

With a sharpened image, the pre-processing steps are complete and the algorithm 

can successfully apply edge detection techniques. This research only compared 
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traditional edge detection methods because they are more computationally efficient while 
maintaining a sufficient level of accuracy. In a comparison of the Sobel method and the 
Canny method, the Sobel operator was the edge detection method of choice due to its 
lesser processing time. The Sobel method uses a 3x3 operator that sums the gradient 
values of orthogonal vectors resulting in a magnitude and direction value for a given 
neighborhood of pixels (Sobel 2014). Within the algorithm, the Sobel operator captures 
all edges within the image that meet a minimum threshold of 0.015, which was 
determined through trial and error. The result is a binary image that only displays the 
lines meeting the threshold set by the Sobel operator. 

The application of the Hough transformation is the most important component of 
the runway detection algorithm. Paul Hough first patented the Hough transform in 1962 
as a means to detect patterns created by subatomic particles in a bubble chamber. The 
Hough transform, as used today, transforms all points in the image frame in a new polar 
parameter space. Within the parameter space, all points in an image frame correspond to 
a sinusoid, points in the parameter space are straight lines in the image, and points on the 
same line in the image frame will share a co mm on point in the parameter space (Hart and 
Duda 1972, 4). The algorithm applies the Hough transform to the edge-detected image 
and searches for lines within the range of -35 and 35 degrees to match the expected 
runway angle from the final approach perspective. The top 5-7 peaks in the Hough 
parameter space are chosen and analyzed in order to determine the best match for the 
runway edges. Typically, the top two peaks are the most likely matches; however, the 
most prominent lines in the image will not necessarily coincide with the edges of the 
runway. To avoid incorrect identification, the algorithm examines the resulting lines for a 
number of common error scenarios. The first error check is the distance between points 
on the runway edges. Any separation of less than 20 pixels indicates that the detected 
runway edges are too close and that incorrect lines have been selected. Therefore, the 
algorithm analyzes the next grouping of peaks in the Hough space. Conversely, if the 
distance between runway endpoints is greater than 150 pixels in the x direction or 40 
pixels in the y direction, the detected edges are too far apart and the algorithm signals a 
line mismatch. Together, these test scenarios ensure that the detected edges fall within the 



expected range for the runway geometry. Additionally, in real-time application, the 
detected runway edges are averaged between image frames, achieving the effect of a low 
pass filter. This limits the effect of mechanical vibrations on the camera and it reduces the 
impact of slight errors within the algorithm. 

The algorithm was tested on 3000 approach images to Monterey Regional Airport 
taken from a TASE200 camera attached to the airframe of a Cessna 206 below the left 
wing. Overall, the system achieved an accuracy of 96.2 percent across all images, 
incorrectly identifying only 114 of the 3000 frames. The primary sources of inaccuracies 
were distance extremes and misidentified clutter within the image. The algorithm 
performs comparatively poorly at extremely long distances and extremely short distances. 
At long distance, noise and blurring in the image make accurate detection of runway 
markings difficult. At short distances, the aircraft is at very low altitudes, which distorts 
the perspective of the runway and makes accurate detection more difficult. Noise and 
clutter only become a problem in images when they align with one of the runway edges. 
If clutter is close enough to the runway edge and large enough to pass through noise and 
clutter elimination methods, it can be misidentified as a part of the runway edge. Outside 
of these specific scenarios, the algorithm performs well. An original runway approach 
image and its corresponding output from the algorithm are shown in Figure 1. 



Figure 1. Original Approach Image (left) and Algorithm Output (right) 


Future applications of this algorithm could include attitude and pose estimation as 
well as complete integration with an aircraft control system. Based on the performance of 
the runway detection and tracking algorithm, it is evident that computer vision could play 
a significant role in the navigation and control of autonomous UAVs. However, much 
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work remains to integrate the algorithm with an aircraft’s control system. For one, 
attitude and pose estimation must be applied to accurately estimate the position of the 
aircraft. This information will play a crucial role in fulfilling the feedback control loop 
within the control system of the aircraft. Overall, it is reasonable to expect that visual 
feedback can play an important role in the future of autonomous aircraft. 
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I. INTRODUCTION 


The introduction presents the current state of unmanned aerial vehicles (UAV) 
and auto-land procedures. This chapter will describe UAVs in a Department of Defense 
(DOD) context while linking the goals of this research to DOD requirements for 
autonomous systems. Additionally, the overall concept and objectives for a vision-based 
landing system and the scope and limitations involved in the research are outlined. 

A. VISION FOR UNMANNED AERIAL VEHICLES 

Autonomous systems are fulfilling increasingly important and complex roles 
within the DOD and foreign militaries across the globe. Not only are autonomous 
systems capable of taking the warfighter out of dangerous situations, but they can operate 
in diverse environ m ents while accomplishing a versatile collection of missions. The DOD 
Unmanned Systems Integrated Roadmap for FY2011-2036 (2011, 3) states, “DOD 
envisions unmanned systems seamlessly operating with manned systems while gradually 
reducing the degree of human control and decision making required for the unmanned 
portion of the force structure.” Thus, the ideal autonomous system is one that can 
accomplish its mission without any human intervention by making tactical or strategic 
decisions without a human in the control loop. While the capabilities of unmanned aerial 
systems (UAS) are increasing, they still require significant levels of human oversight and 
external control inputs, usually via remote piloting at ground control stations. In order to 
achieve increased autonomy in DOD systems, decision-making and control processes 
must be integrated into the system itself The advanced autonomous vehicle should be a 
self-contained system capable of gathering information from its surroundings, processing 
the information relative to its tactical scenario and mission, determining a course of 
action, and acting on its decision without external human inputs. 

The DOD Research and Engineering Technical Assessment on Autonomy (Office 
of Technical Intelligence 2015, 4) encourages additional research into low-cost systems 
capable of achieving autonomous “perception, cognition, and action.” The statement 
implies an underlying intent to develop unmanned systems that fulfill a variety of 
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dangerous and complex roles while doing so at a lower cost than current mission 
execution levels. Achieving full autonomy in aerial vehicles is technologically 
challenging and the solutions tend to be expensive. This research offers a method to 
increase autonomy in UAVs via low-cost vision systems. While this research will focus 
on aerial systems, the basic principles of vision-based control are applicable to all 
autonomous systems in the DOD. 

B. CURRENT STATE OE MANNED AND UNMANNED AIRCRAET 
LANDING PROCEDURES 

This section provides a brief overview of both manned and unmanned aircraft 
landing procedures. The first subsection focuses on general landing procedures while the 
second and third subsections focus on the technical aspects of assisted and autonomous 
landings. 

1. General Landing Procedures 

This research focuses on the stages of landing from final approach to touchdown 
of the aircraft. Specific landing procedures will vary based on the size and aerodynamic 
characteristics of the aircraft and environmental conditions; however, the Federal 
Aviation Administration outlines a general approach to landing in its Airplane Flying 
Handbook. The pilot reaches the final approach when the heading of the aircraft, given no 
crosswind, is aligned with the center of the runway and the aircraft is following the 
desired glide slope (Federal Aviation Administration [FAA] 2016a, 8.3). Throughout the 
final approach, the pilot should make constant adjustments to align the aircraft with the 
runway centerline, maintain the appropriate glide slope, and maintain the appropriate 
airspeed via control inputs to the rudder, ailerons, elevators, flaps, and engine power 
(FAA 2016a, 8.3-8.6). The pilot should aim for a predetermined touchdown point 
throughout the final approach, typically delineated by two specific markings on either 
side of the runway beyond the threshold (FAA 2016a, 8.10). A useful tool for the pilot in 
achieving a consistent approach angle is to ensure that runway perspective remains the 
same (FAA 2016a, 8.10). An elongated, narrow perspective indicates a steeper glide 
slope while a flatter, shorter perspective indicates a shallow glide slope (FAA 2016a, 
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8.10). This technique should be continuously applied until the pilot must flare the nose of 
the aircraft and touchdown. The common theme throughout the landing process is that 
visual cues are essential to maintaining an accurate final approach and completing a 
successful landing. By using a vision-based landing procedure for autonomous systems, 
the same general procedure applies. 

2. Assisted Landings 

A common and well-accepted landing navigation aid is the Instrument Landing 
System (ILS). The system uses components called a localizer and a glide slope to provide 
incoming aircraft with azimuth and elevation information for landing (FAA 2016b). The 
localizer generates VHP signals that the incoming aircraft can translate into relative 
positions left or right of the runway centerline (FAA 2016b). The glide slope station also 
generates VHF signals that the incoming aircraft translates into elevation positions either 
above or below a three-degree angle of descent (FAA 2016b). When carefully integrated 
with runway markings and lighting, ILS becomes an integral tool for pilots. 

Distance Measuring Equipment (DME) is another navigational aid that is often 
integrated with ILS. DME stations at airports receive and relay radio signals from 
incoming aircraft, which the aircraft’s DME equipment can then use to calculate distance 
from the runway (FAA 2014). DME differs from localizer and glide slope stations 
because it provides direct slant range between the runway and the aircraft rather than 
azimuth and elevation information (FAA 2014). 

Naval aviators use an improved version of the Fresnel Lens Optical Landing 
System (FLOLS) as a navigation aid for landing onboard aircraft carriers (Golovcsenko 
1976, 9-11). As with ILS, FLOLS provides the incoming aircraft with its position 
relative to the center of the runway through a lighted reference station located at the edge 
of the flight deck (Golovcsenko 1976, 9). The pilot sees a bar of light at the center of the 
FLOLS station, when the bar is stationed above the green reference lights on either side, 
the glide slope is too steep and when the bar is stationed below, the glideslope is too 
shallow (Golovcsenko 1976, 9). The FLOLS is depicted in Figure 1. 
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Figure 1. FLOLS System Onboard an Aircraft Carrier. Source: NAVAIR Fleet 

Readiness Center Southwest (2003). 

3. Current Autonomous Landing Capability 

Few unmanned aerial vehicles can successfully takeoff and land autonomously. 
Most notable are the RQ-4 Global Hawk and the X-47B, both produced by Northrop 
Grumman. The RQ-4 Global Hawk is an operational unmanned aerial system designed 
for intelligence, surveillance, and reconnaissance with over 200,000 operational flight 
hours (Northrop Grumman 2017). The X-47B is an unmanned combat air system 
(UCAS) created to test the feasibility of an autonomous carrier-based air platform 
(Northrop Grumman 2015). Together, these aircraft represent the future of UAVs as used 
by the U.S. military while leading the field in auto-land technology. 

The autonomous landing process for both aircraft is similar. Both use differential 
global positioning systems (DGPS) and inertial navigation systems (INS) to make 
precision adjustments in the landing phase. DGPS uses a stationary radio transmitter in 
addition to the four satellites necessary for a GPS fix, significantly increasing the 
accuracy of the system (Defense Standardization Program 2010, 5). The Global Hawk 
specifically uses dual KN-4072 INS/GPS onboard systems for guidance in the landing 
stage (Loegering and Harris 2002, 2). Prior to takeoff and mission execution, the Global 
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Hawk is loaded with a series of GPS waypoints for the landing sequence that guide the 
aircraft to the runway and indicate when the aircraft should begin executing specific 
landing procedures (Loegering and Harris 2002, 3). For example, a waypoint positioned 
approximately 150 seconds from the point of touchdown indicates that the Global Hawk 
should change speed to meet final approach requirements (Loegering and Harris 2002, 3). 
Though not an exact replication, the X-47B utilizes a similar procedure that closely 
mirrors the Joint Precision Approach and Landing System (JPALS). JPALS sea-based 
systems, such as those used on aircraft carriers, implement a landing process similar to 
that of the Global Hawk, but they also integrate data from a shipboard INS to account for 
the continuous motion of the ship (Defense Standardization Program 2010, 6). This 
additional layer of INS data allows incoming aircraft such as the X-47B to land with the 
accuracy and precision required for carrier-based operations. DGPS-based systems, such 
as those found on the RQ-4 and the X-47B, offer superior performance for autonomous 
landings, but they add a significant level of complexity to the system. 

C. VISION-BASED LANDING SYSTEM CONCEPT 

Developing a vision-based landing system requires three central components: an 
image processing algorithm, a pose estimation algorithm, and a control system integrated 
with the unmanned aerial vehicle. 

1. Control 

The execution of the vision-based landing system will follow the framework for a 
standard control loop, as shown in Figure 2. Error signals related to heading, altitude, and 
speed will serve as inputs to the controller, which calculates corresponding outputs for 
the aircraft control surfaces to achieve the desired state. In this case, the desired state is a 
heading aligned with the center of the runway, a three-degree angle of descent, and 
consistent approach speed. The resulting state of the aircraft following these control 
actions will be measured and fed back into the controller via the vision-based system 
established in the previous steps. The camera will capture an image of the runway, which 
the pose estimation algorithm will then use to estimate the attitude and position of the 
aircraft and consequently the error signal for the controller. The unmanned system will 
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continuously execute the eontrol loop until the aircraft safely lands. In Figure 2, ea, eh, 
and es represent the altitude, heading, and speed error, respectively. Likewise, 6e, 6r, and 
6t represent the required changes in elevator angle, rudder angle, and thrust, respectively. 



Figure 2. Vision-Based Landing Control System 

2. Image Processing 

The image processing eomponent of the vision-based landing system involves 
detection and tracking of the runway in images taken from a camera onboard the aireraft. 
The objective of the image processing component is to accurately identify and localize 
the runway while distinguishing relevant markings and characteristics of the runway. The 
various eomputer vision techniques that allow for the sueeessful completion of this 
step will be outlined in later chapters, as the image processing component is the focus 
of this research. 

3. Pose and Attitude Estimation 

Pose and attitude estimation is the process of ealculating the roll, piteh, yaw, glide 
slope, and speed values for the aireraft. This is the stage in the vision-based landing 
system in which the system gains situational awareness by analyzing runway information 
provided by the previous image proeessing algorithm. Thus, the same visual cues used by 
human pilots can be used by unmanned autonomous systems. 
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a. Heading Error 

Heading error is crucial to maintaining alignment with the centerline of the 
runway in the final approach stage. The system can maintain proper heading by ensuring 
that the internal angles of each edge of the runway are equal, assuming that the position 
of the camera in relation to the center of the aircraft is known and that the aircraft is not 
experiencing any sideslip. Equation 1.1 describes the relationship between internal 
runway edge angles and the heading error. 

e,=9,-9, + f{/3) (1.1) 

In Equation 1.1, the theta values represent the left and right internal angles, 
respectively, while the final value represents a function that adjusts for the sideslip angle 
of the aircraft depending on the aircraft dynamics and external conditions. 

b. Altitude Error 

The algorithm should calculate altitude error as a ratio of the distant runway edge 
to the leading runway edge, assuming that the length and width of the runway is known. 
If the aircraft is maintaining a consistent glideslope through descent, the ratio will remain 
the same. Larger or smaller ratios between the runway ends should trigger an error signal 
in the system, as these conditions indicate the glide slope is either too shallow 
or too steep. 


c. Speed Error 

Estimating speed from vision alone is a difficult task and it is not one that human 
pilots can accurately complete. Therefore, it is reasonable to assume that some assistance 
may be required from additional sensors, such as pitot tubes, in order to calculate 
airspeed. Visual cues that may lend to the calculation of airspeed could include the 
calculation of the rate of change of a known distance in the image frame, such as the 
width of the runway. 
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D. RESEARCH QUESTION AND OBJECTIVES 

A capability need statement for future UAVs undoubtedly evokes the need to 
create an accurate, robust, and cost efficient system that can successfully achieve a given 
mission profile without human intervention. While achieving autonomy in aerial systems 
is a multi-faceted issue, developing an improved autonomous landing capability remains 
a critical issue for the future of unmanned systems. What is required is a systems 
engineering approach to the development and assessment of autonomous landing 
processes. While the RQ-4 and X-47B have demonstrated autonomous landing 
capabilities, they are complex, expensive, and still require some degree of human 
intervention. A critical component of systems engineering is the process of analyzing the 
solution space to find the best approach to overcome a given problem. The process must 
include trade-off and feasibility analysis to determine the best solution for all 
stakeholders. This thesis uses a systems engineering approach to evaluate the feasibility 
of a real-time, vision-based runway detection and tracking algorithm that will aid in the 
landing of a fixed-wing UAV. As this is a complex problem, it requires in-depth analysis 
and a fundamental understanding of the technical issues. This research must first explain 
the framework and methodology for the vision-based landing system, then conduct 
objective analysis of its performance and feasibility in future systems. Thus, this thesis 
aims to provide a quantitative and qualitative analysis of a computer vision approach 
intended to achieve an autonomous landing capability. 

E. SCOPE, LIMITATIONS, AND ASSUMPTIONS 

A satisfactory runway detection and tracking algorithm should be extremely 
accurate so as to prevent false detection and, therefore, the passing of erroneous signals to 
the aircraft control system. The algorithm should also be robust and reliable with the 
ability to detect the location of the runway at long and short ranges, in all conditions 
meeting visual flight rule (VFR) weather minimums as defined by the Federal Aviation 
Administration (FAA), and through noisy image frames taken from live video onboard 
the aircraft. Processing time must also be a consideration for eventual application in a 
real-time environment. Fast processing times are often at odds with increased accuracy 


8 



and robustness. Therefore, one must evaluate every aspect of the detection and tracking 
process based on necessity and optimization. 

The fields of unmanned autonomous vehicle research and computer vision are 
vast, highlighting the importance of scope limitation for individual research projects. This 
thesis focuses on autonomous fixed-wing aircraft. Size of the aircraft is not necessarily a 
relevant factor so long as the characteristics of the aircraft do not call for special 
considerations for control during the final approach and landing stages of flight. 
Therefore, the results of the detection and tracking process for fixed-wing aircraft may 
still be relevant to all unmanned aerial vehicles including rotorcraft. 

While the computer vision algorithm should be robust for use on a wide variety of 
runways, a few general assumptions are required. First, the runway markings used as 
visual cues should abide by Federal Aviation Administration standards as outlined in AC 
150/5340-IL also titled “Standards for Airport Markings.” Secondly, all runways should 
have a centerline and side stripes for accurate detection. These are standard markings for 
precision instrument runways. Lastly, there should be no obstacles or obstructions in line 
of sight to the runway in the final approach stage of flight. A runway meeting all of these 
assumptions is depicted in Figure 3. 



Figure 3. Precision Instrument Runway and Markings in Accordance with 
Federal Aviation Administration Standards. Source: FAA (2015). 
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II. DATA SAMPLES AND LITERATURE REVIEW 


This chapter describes the data used for evaluation of the vision-based runway 
detection and tracking algorithm. A complete literature review and thesis overview is also 
included to provide reference frame for the reader. 

A. DESCRIPTION OF AVAILABLE DATA 

The images used for this research feature a final approach at Monterey Regional 
Airport (MRY). The data was collected from a Cessna 206 airplane with a TASE200 
camera attached to the airframe below the left wing. The setup is shown in Figure 4. 



Figure 4. Aircraft and Camera Setup for Collection of Data. 

Source: Gloria (2016). 


The camera has a frame rate of 30 frames per second (fps) and this research used 
3000 frames of the final approach video. The remaining characteristics of the TASE200 
camera are shown in Table 1. 
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Table 1. Specifications for TASE200 Camera. Adapted from Gloria (2016). 


MECHANICAL SPECIFICATIONS 

Diameter 

4.4 inches 

Height 

7.5 inches 

Weight 

2.34 pounds 

PERFORMANCE 

Use 

Daylight and infrared imaging 

Pan limits 

360° continuous 

Tilt limits 

+237-203° 

IR camera 

Resolution:640x480; HFOV: 10.5° 

Daylight camera 

Optical zoom: 3lx; HFOV:55.7°-1.94° 


The images cover approximately one minute and 40 seconds of flight time in the 
final approach to the runway at Monterey Regional Airport. The starting altitude is 
approximately 1461.0 feet and the final altitude matches the elevation of Monterey 
Regional Airport’s runway. The average airspeed throughout the descent was 
approximately 94.5 knots. Samples of the runway images from long, medium, and short 
range are shown in Figure 5. 



Figure 5. Fong (left). Medium (middle), and Short (right) Range Approach 

Images to Monterey Regional Airport 


The TASE200 system also collected INS and GPS metadata from the flight that 
corresponds to each individual image frame. The GPS data from the final approach to 
Monterey Regional suggests a relatively straight flight path, as shown in Figure 6. The 
blue line represents the path of the aircraft while the red points represent the estimated 
target area of the camera throughout the final approach. 
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Figure 6. Flight Path Represented in Latitude and Longitude Coordinate Frame. 

Adapted from AirNav (2017). 

The system also captured altitude, speed, and attitude information from the 
aircraft for each individual image frame. The altitude and heading data is shown in Figure 
7. It is apparent that the aircraft follows a relatively stable rate of descent and maintains a 
consistent heading. In Figure 7, the blue line represents the aircraft altitude and the red 
line represents the altitude of the camera’s target. 
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Figure 7. Altitude and Heading Flight Data 
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For future pose estimation evaluation, it will be important to consult the true roll, 
pitch, and yaw data from the flight. This data is displayed in Figure 8, where the blue and 
red lines represent the position of the TASE200 camera gimbal and mount, respectively. 
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Figure 8. Roll, Pitch, and Yaw Flight Data 


Additionally, the TASE200 system collected speed data from the flight. The 
corresponding information is shown in Figure 9. The top graph displays the aircraft speed 
calculated using GPS data while the bottom graph shows speed relative to the vertical 
axis of the body of the aircraft. 




Figure 9. Speed Flight Data for GPS Speed (top) and 
Vertical Axis Speed (bottom) 
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Finally, it may be necessary to calculate transformations from the camera frame 
to the aircraft body frame or the world frame. These calculations would require 
information concerning the pose of the camera in each frame. This information is 
displayed in Figure 10. 
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Figure 10. Roll, Tilt, and Pan (left) and Roll, Tilt, and 
Pan Rate (right) Flight Data 


B. LITERATURE REVIEW 

Research in the fields of control theory and computer vision has resulted in a 
number of methods recommended for real-time runway detection and tracking. This 
section reviews previous research that provides the foundation for this thesis. Based on 
these techniques and methodologies, this thesis seeks to improve detection and tracking 
performance and present alternatives to improve accuracy or processing time. Broadly, 
the relevant research areas include runway detection and tracking, pose and attitude 
estimation, and control theory. 

In “Vision-based Runway Recognition for UAV Autonomous Landing,” Jiajia 
Shang and Zhongke Shi (2007) describe one of the most complete methods for runway 
detection and implementation in an aircraft control system. Generally, Shang and Shi’s 
methodology includes image preprocessing, runway area location, edge detection, and 
runway edge determination. The preprocessing stage uses a 3x3 crisscrossing median 
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filter, a standard and reliable image processing technique, to eliminate noise in the image 
(Shang and Shi 2007, 112). The runway location stage uses a double grayscale threshold 
technique that eliminates pixels below a predetermined threshold from an image, leaving 
only those regions in the image that lie on the runway (Shang and Shi 2007, 113). 
Grayscale thresholding reduces processing time because it only analyzes intensity 
information. While quicker, it also eliminates hue and saturation information that might 
contribute to the accuracy of detection. Shang and Shi chose to use the Sobel operator to 
detect vertical and horizontal lines in the image for further processing (2007, 113). The 
Sobel operator uses vector summations from a 3x3 neighborhood to gather gradient 
information for each pixel (Sobel 2014). Following edge detection, Shang and Shi used 
the Hough transform to find lines in the parameter space that correspond to the runway 
edges (2007, 114). This process is well suited for accurate real-time detection of runway 
edges and this thesis uses the general framework of this algorithm. 

Other common runway detection methods involve matching real-time images to 
templates of a runway to determine distance and position. This method is typically more 
time intensive and prior images of the runway are required for successful detection and 
tracking. While this technique may be feasible for UAVs that consistently use the same 
runway for landings, it does not meet the requirement of robustness as previously 
discussed in this chapter. 

C. THESIS ORGANIZATION 

As previously stated, this thesis specifically focuses on the image processing 
component of the larger vision-based landing system, which aims to accurately detect and 
track a runway in real time. The systems engineering feasibility analysis starts with a 
complete description of the framework and methodology of the runway detection and 
tracking algorithm in Chapter Iff. This chapter outlines the pre-processing steps necessary 
to localize the runway and eliminate noise and clutter. Next, Chapter fV reviews the 
chosen edge detection technique, the Hough transformation to identify the runway edges 
within the image, and the error detection and correction methodology. Then, Chapter V 
outlines the result of the feasibility analysis and includes a discussion of the algorithm’s 
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performance. This thesis ends with Chapter VI presenting conclusions from this research 
and outlines future work for autonomous landing system development. 
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III. FRAMEWORK AND METHODOLOGY: 
PRE-PROCESSING 

This chapter describes the detailed pre-processing steps of the runway detection 
algorithm. These steps include image thresholding, morphological operations, image 
masking, and image sharpening. The algorithm was developed and executed in 
MATLAB. Use of any predefined functions from the MATLAB image processing 
toolbox is explicitly stated. 

A. THRESHOLDING DESCRIPTION AND PURPOSE 

Thresholding is the process of segmenting an image based on a specific 
characteristic such as color or intensity (Cheng, Sun, and Wang 2011, 28). Thresholding 
is a fundamental image processing technique because it is a simple and computationally 
efficient way to focus on relevant areas while eliminating irrelevant regions. 
Thresholding results in the isolation or elimination of a specific region within the image 
for further processing. While grayscale intensity thresholding is a common technique, all 
color spaces are can be used for thresholding but determination of the correct color space 
requires trade-off analysis for accuracy and processing time. 

The purpose of image thresholding for runway detection is twofold. First, it is the 
initial step for localizing the runway within the given frame. With properly defined 
thresholds, the algorithm should effectively highlight the area of interest, which is the 
runway. Additionally, by highlighting the relevant areas of an image, thresholding should 
save time in later stages of the algorithm by limiting the number of pixels processed 
through more complex and computationally intensive steps. Based on the results of this 
research, it is evident that thresholding often will play the greatest role in eliminating 
noise and clutter from the image. While thresholding has the potential to eliminate large 
unnecessary areas, it is still important to avoid elimination of important data within the 
image when constructing the thresholding limits. 
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1. Description of Color Spaces 

Alvy Ray Smith introduced the HSV (hue, saturation, and value) color space in 
1978. Smith (1978, 3) described hue as “the dimension with points on it normally called 
red, yellow, blue-green, etc.,” saturation as “the departure of a hue from achromatic, i.e., 
from white or gray,” and value as “the departure of a hue from black.” Hue, saturation, 
and value can describe colors in a more intuitive fashion. Smith (1978, 4) likened the 
HSV color model to the process used by an artist to create his or her paint, where he or 
she would choose a base color and add white or black paint in order to achieve the 
desired saturation and value. Smith’s primary objective in creating the HSV color space 
was to provide an alternative to the RGB (red, green, blue) color space, which was widely 
used but difficult to conceptualize. A visual representation of the conical HSV space is 
shown in Figure 11. 



Figure 11. Conical Representation of the HSV Color Space 

Electronic devices and monitors typically use the RGB color space to produce and 
display images. While there are variations of the RGB model, they all use some 
combination of red, green, and blue channels to create the full spectrum of colors. The 
chromaticity of a color defines this red, green, and blue combination and the mixing 
process is a close reproduction of how the receptors in the human eye process colors via 
wavelength combinations (Joblove and Greenberg 1978, 20-21). 


20 







Grayscale images eliminate all hue and saturation information, leaving intensity, 
or value, as the only remaining eharaeteristic. All pixels within a grayseale image 
will therefore earry some value at or between the extremes of blaek and white. The 
simplieity and intuitive nature of grayseale images make them popular for image 
proeessing. Grayseale images also provide an advantage in proeessing speeds, as the 
images will only contain one channel for intensity compared to three ehannels in the HSV 
and RGB color space. 

2. Comparison of Color Spaces 

Determining the best color space for the thresholding proeess required a thorough 
comparison based on accuracy and computation time metries. The three options were the 
RGB color space, the HSV eolor spaee, and intensity for grayseale images. 

The RGB eolor spaee was eliminated early in the eomparison proeess beeause 
runway surfaees tend to include little variation in chromatieity, making it difficult to 
achieve any degree of aeeurate thresholding using red, green, and blue ehannels. Red, 
green, and blue ehannel thresholding is most effeetive when the target or area of interest 
is a unique color and its particular combination of red, green, and blue allows for distinct 
separation from the surrounding environment. Unfortunately, this is not the case for the 
dark runway surfaces and white runway markings, which only fall at the extreme values 
of the red, green, and blue channels. If anything, RGB analysis demonstrated the 
importanee of using multiple image characteristics for analysis rather than simply 
chromatieity or intensity. 

After comprehensive eomparison, research and experimentation favored the HSV 
color space over the grayseale color space. Implementation in the MATLAB environment 
proved that the grayseale thresholding process is aetually quite involved. The RGB color 
space is the most eommon choiee for image storage on eleetronie devices, so the 
algorithm must first eonvert the image into the grayscale eolor spaee from the RGB 
spaee. Following this eonversion, eaeh pixel in the image must undergo a gradient 
calculation in order to determine the maximum and minimum intensity values for the 
thresholding limits. It is possible to choose predefined threshold limits that do not use a 
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relative gradient relationship, thus eliminating the need for the gradient operation. 
However, this has a decidedly unacceptable effect on accuracy, as intensity changes 
considerably with distance from the runway, environmental conditions, and noise. 
Experimental results comparing HSV and grayscale thresholding times support the 
conclusion that HSV thresholding is actually the superior method. A test of 30 trials 
for each method determined that the difference between HSV and grayscale mean 
processing times was 0.147 seconds ±0.010 seconds in favor of the HSV color space (see 
the Appendix for full results). Considering real-time requirements for the algorithm, 
0.147 seconds is a significant difference. 

The HSV color model was also the preferred choice in accuracy comparisons. The 
inclusion of hue and saturation data in the thresholding process resulted in a more robust 
controller that could accurately localize the runway region through a wider range of 
distances, varying environmental conditions, and image noise levels. When only working 
with intensity values, the user is limited to the manipulation of one channel for 
thresholding. Wide ranges will eliminate noise and clutter but may result in the 
elimination of relevant areas in the runway region. Narrow threshold ranges will include 
all the relevant areas of the runway but increase noise and clutter in the output image. 
The difference between these extremes is small but the results are drastically different, as 
shown in Figure 12. 



Narrow intensity threshold from 0.65 to 0.85 (left) compared to a wider intensity 
threshold from 0.60 to 0.90 (right). 

Figure 12. Comparison of Narrow and Wide Intensity Threshold Margins 

for a Grayscale Image 
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It is evident in Figure 12 that choosing threshold limits is a sensitive process. The 
intensity ranges only varied by a value of 0.05 for the upper and lower limits, but the 
results are dramatically different. Considering the uncontrollable factors that can affect 
image intensity, relying on a single image characteristic to achieve robustness in the 
thresholding process is challenging at best. In the HSV color space, the combination of 
hue, saturation, and value channels adds redundancy in the thresholding process, 
improving its resilience to variation due to uncontrollable factors. Therefore, HSV 
thresholding was the more accurate method in almost all trials. 

3. Applied HSV Threshold Limits 

The most appropriate approach to determining the upper and lower HSV 
threshold limits was through trial and error. The greatest challenge was finding limits that 
worked at both long and short distances from the runway. Extensive analysis of 3000 
frames of approach images to Monterey Regional Airport and incremental adjustments to 
the upper and lower limits revealed the appropriate threshold range, as defined in 
Equation 1.2. 

Hue = (0 < h < 0.440) u (0.700 <h<\) 

Sat =(0<5<0.200) (3.1) 

Val = (0.637 < V < 1) 

The hue range includes all portions of the color wheel except those in the blue to 
cyan range. Excluding these hues is an efficient way to eliminate clutter and noise 
commonly resulting from the sky. Future applications in aircraft control may use the sky 
and the horizon line as aids to determine attitude and elevation, but these aspects are 
irrelevant to the runway localization and detection process. 

The saturation limits range from 0.00 to 0.200 because runway markings are 
always white or light gray, which correspond to lower saturation values. The runway and 
taxi surfaces are made from concrete and asphalt, which will generate lower saturation 
values in images, thus aiding the localization process. This tends to create contrast with 
the surrounding environment, which usually includes saturated colors found in trees, 
grass, and other common airport environments. 
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The value limits favored brighter colors that highlight the white runway markings. 
For runways with black asphalt surfaces, the distinction between the markings and the 
runway surface is clear, making value thresholding even more useful. The distinction 
between markings and the surface is less clear on runways with lighter concrete surfaces, 
which slightly dilutes the effect of value thresholding. 

Thresholding effectiveness is lowest at either extreme, both long and short, with 
performance at mid-range proving to be the most accurate. A comparison of long, 
medium, and short-range thresholding is illustrated in Figures 13 andl4. 



Figure 13. Original Approach Images at Long, Medium, and Short Range 



There is a large deviation in the results of the thresholding process. In some cases, 
thresholding will only capture the runway markings. In other cases, it will capture the 
entire runway as well as portions of the environment surrounding the runway. Both 
scenarios are acceptable as long as the thresholding process generally localizes the 
runway and eliminates large portions of unnecessary information from the image. Ideally, 
the resulting binary image will only include the runway markings on and around the 
runway, but an exact solution at all ranges and in all environments is impossible to 
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achieve. The next step, which applies morphological operations, will work to normalize 
the result of the thresholding process. 

B. MORPHOLOGICAL OPERATIONS 

This section provides a brief description of the purpose of morphological 
operations and an overview of the primary operations and functions used in the 
algorithm. 

1. Background and Purpose 

Morphology aids in the identification and analysis of shapes within an image 
(Dougherty and Lotufo 2003). The objective of image processing is often to identify or 
preserve some boundary, area, or region, which makes morphology invaluable to the field 
of computer vision. Some of the first uses of morphology date back to experimentation 
by Kirsch, Cahn, Ray, and Urban (1957), in which computer code completed several 
image pre-processing steps, removing humans from time-intensive operations. From 
those foundations, the process and purpose behind morphology largely remains the same. 
In the context of this research, the purpose of morphology is to further localize the 
runway area and capture the outline of the runway and any markings on the surface. 

2. Erosion and Dilation 

The two basic morphological operations are erosion and dilation. Traditionally, 
algorithms use these operations on binary images to identify shapes and regions more 
accurately. Working with binary images also decreases complexity as matrix operations 
only include ones and zeros. 

Dilation is the process of enlarging a shape or region within an image by 
expanding the boundary of that shape (MathWorks 2017i). The number of pixels used to 
enlarge the region depends on the structuring element used in the dilation process 
(MathWorks 2017i). For MATLAB functions, the dilation rule states, “the value of the 
output pixel is the maximum value of all the pixels in the input pixel’s neighborhood” 
where the output pixel and neighborhood pixels are determined by the shape and size of 
the structuring element (MathWorks 2017i). Dilation can recapture portions of an image 
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that may have inadvertently been lost in pre-processing steps. An example of two shapes 
dilated by a 30x30 square structuring element is shown in Figure 15. 



Figure 15. Example of Dilation Operation with 30x30 Square Structuring Element 

Erosion is the contrasting morphological operation, where the boundary of a 
region is reduced, thus shrinking the area of the shape (MathWorks 2017i). In this 
scenario, the minimum value within a neighborhood of pixels defined by a structuring 
element determines the value of the output pixel (MathWorks 2017i). Erosion eliminates 
noise and unwanted clutter in an image. An example of the erosion operation with a 
30x30 pixel square structuring element is shown in Figure 16. 
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3. Opening and Closing Operations 

Opening and closing operations combine the erosion and dilation processes to 
improve the image while preserving the shape and boundary of the objects of interest. 
This is specifically useful when the objective is elimination of noise and clutter. 

The opening operation is the sequential application of erosion followed by 
dilation. Erosion will eliminate smaller, unwanted binary regions and dilation will restore 
the region of interest to its original size. An example of clutter elimination is shown in 
Figure 17, with a square structuring element used to preserve the square shape of the 
region of interest. 



Figure 17. Opening of an Image with 20x20 Square Structuring Element 

to Eliminate Clutter 

Closing involves the sequential application of dilation and erosion. The most 
common use for the closing operation is to remove gaps or holes in a region of interest. A 
simple example of this application is illustrated in Figure 18, where a 20x20 square 
structuring element fills gaps in a rectangular region. 
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Figure 18. Closing of an Image with 20x20 Square Structuring Element to Fill Gaps 

4. Application in Runway Detection Algorithm 

For demonstration purposes, a final approach image taken approximately 
37 seconds from touchdown will demonstrate the result of morphology in the context of 
this algorithm. The original image and thresholded image are shown in Figure 19. 



Figure 19. Original and Thresholded Image for Use in Morphological Operation 

Demonstration 

The first operation was a filling operation available in MATLAB’s image 
processing toolbox labeled “imfill” (MathWorks 2017a). The function acts as a closing 
operator, filling in gaps or spaces in enclosed regions within a binary image. The result of 
this step is illustrated in Figure 20. 
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Figure 20. Image Produced after Executing “imfill” Function in MATLAB 

With all holes filled in the binary regions of the image, the next step is to 
eliminate noise and clutter. An opening operation is used to eliminate the smaller clusters 
of pixels that passed through the threshold process but do not contribute to runway 
localization. MATLAB’s “bwareaopen” function, also included in the image processing 
toolbox, was used to accomplish this task (MathWorks 2017b). This function uses the 
opening operation to eliminate pixels below a defined threshold area. The runway 
detection algorithm uses an area threshold of 30 pixels for the opening operation, which 
was determined through a process of trial and error. Thirty pixels will eliminate most 
noise but retain all areas on or near the runway. The final product of the “bwareaopen” 
operation is shown in Figure 21. 




It is apparent in Figure 21 that some of the markings on the runway surfaee 
were lost in the morphology process. This is acceptable because the primary objective 
is to identify the runway edges and later stages of image processing can recapture 
lost features. 

Dilation is the next morphological operation applied to the binary image. The 
objective of this final step is to recapture any area of the runway that were lost or 
eliminated in previous steps. The dilation operation uses a vertical, rectangular 
structuring element, as runway markings and components are typically vertically oriented 
rectangles. The result of this process is illustrated in Figure 22. 



Figure 22. Image Produced after Executing Dilation with 10x4 Vertical 

Structuring Element in MATLAB 

Ideally, dilation will capture the complete outline of the runway with no breaks or 
gaps, as shown in Figure 22. However, the algorithm will occasionally fail to capture 
every portion of the runway edge and surface. A robust design and redundancy 
throughout the algorithm will allow later steps to either correct the mistake or overcome 
the lack of data. 

The final operation is a second iteration of the “imfill” function, which refines the 
morphology output (MathWorks 2017a). If the algorithm identifies the entire perimeter of 
the runway without gaps or breaks, the second filling operation will fill the interior 


30 


surface of the runway, an added benefit for the creation of an image mask. The result of 
the second iteration of the “imfiH” function is shown in Figure 23 (MathWorks 2017a). 



Figure 23. Image Produced After Executing Second Iteration 
of “imfill” Function in MATLAB 

C. IMAGE MASKING AND FILTERING 

This section covers the masking and filtering steps in the runway detection 
algorithm. Included are descriptions of each process, the purpose and justification for 
each step, and a brief overview of the types and sources of noise common to images. 

1. Image Masking 

Image masking is a process that captures a desired portion of an image while 
setting all other background pixels in the image to zero, or black. For the runway 
detection algorithm, the resulting binary image from the morphology process serves as 
the mask for the original runway image. Filtered sections of the original runway image 
will replace any white sections of the binary image that remained after thresholding and 
all morphological operations. In other words, image masking merges the binary image 
with the filtered version of the original image. This concept is best illustrated by Figure 
24, in which the final masked image is displayed. 
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Figure 24. Image Mask of Binary Image and Median Filtered Runway Image 


Aside from a few areas of clutter, the final product should include the runway 
region with as many markings visible on the surface as possible. An accurate image 
mask will lead to improved edge detection and line detection via the Hough transform at 
later stages. 

Naturally, the algorithm will not always work perfectly and portions of the 
runway will be missing from the final image mask. However, this should result in failure. 
Even with partial data, the algorithm can achieve accurate detection of the runway edges. 
Designing a robust detection algorithm requires that varying degrees of accuracy must be 
acceptable at the image mask stage. Multiple examples of less accurate image masking 
results are shown in Figure 25. All of these images successfully resulted in accurate 
runway detection. At minimum, the image mask should at least include the edges of the 
runway, from start to finish, without breaks. 



Figure 25. Examples of Various Levels of Image Mask Accuracy 
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2. Image Filtering 

Image filtering is a pre-processing technique to eliminate noise in an image. There 
are a number of approaches to filtering, each effective at handling various levels and 
types of noise. Choosing an appropriate filter involves analyzing the type of noise 
affecting an image and determining acceptable computation time. Specifically, this 
research considered the effects of three types of noise: Gaussian, salt and pepper, and 
quantization noise. 

Gaussian noise is random additive noise that follows a Gaussian distribution and 
affects all pixels in an image (Dangeti 2003, 6). Internal components and processes of the 
camera, such as amplification, typically introduce Gaussian noise (Patidar, Gupta, 
Srivastava, and Nagawat 2010, 46). Due to its pervasiveness, all image processing 
algorithms should account for Gaussian noise. 

Salt and pepper noise is a series of intensity spikes that randomly affect an image 
(Dangeti 2003, 7). The affected pixels will assume either a maximum or a minimum 
value as a result of malfunctions in the sensors of the camera or errors in the analog-to- 
digital conversion process (Dangeti 2003, 7). The name is derived from the salt and 
pepper spikes that are evident in the final image as the noise results in either white or 
black pixels. Again, salt and pepper noise is a common issue in image processing and 
computer vision so efforts should be made to reduce its effects. 

Quantization noise occurs during image compression, which is required for JPEG 
images. The final approach images used for this research were stored in JPEG format in 
order to store a vast quantity of images; therefore, quantization error was a concern. The 
conversion process compressed 8x8 blocks of pixels resulting in the loss of information 
and a subsequent blocking effect in the image (Quijas and Puentes 2014, 1). The more the 
image is compressed, the more apparent the blocking effect (Quijas and Puentes 2014, 1). 

There are many image filtering techniques available and the techniques grow in 
number and complexity as the field of image processing advances. This research 
considered two filtering methods, the mean filter and the median filter. These time-tested 
and proven techniques provide solutions to a number of image processing issues and they 


33 



are relatively simple in application. Effectiveness and computation time were the 
evaluation factors for each filter. 

The mean filter averages the intensity values of a predefined window of pixels 
and replaces the center pixel, the pixel of interest, with that average value (Dangeti 2003, 
12). This results in a smoothing or blurring effect across the image that eliminates 
random noise. The result depends on the severity of the noise as well as the size of the 
chosen filter. Larger filters will result in more smoothing or blurring (Dangeti 2003, 12). 

Median filtering works in much the same way but it is a nonlinear process 
(Dangeti 2003, 18). The median filter also uses a predetermined window size to evaluate 
neighboring pixels, but the median value from the neighborhood replaces to center pixel 
of interest (Dangeti 2003, 18). This also has a blurring effect on the image. The 
advantages of the median filter are twofold. First, it is more robust in the sense that it 
eliminates outliers, as is true of any process that uses medians instead of means (Dangeti 
2003, 19). Many also argue that the median filter is better suited for edge preservation 
because it must use the value of an actual pixel from the neighborhood (Dangeti 2003, 
19). As there is not much difference between computation time for the mean and median 
filter, these advantages make it the better choice for application in the runway detection 
algorithm. Runway edge detection is the primary goal, so edge preservation should be a 
valued characteristic for a filter. 

To reduce processing time, median filtering should apply only to the image mask, 
so only select regions of the image require computation. However, selective application 
of the filter is difficult and time intensive in MATLAB, so the algorithm filters the full 
original image prior to masking for proof of concept. The result of applying the 
“medfilt2” MATLAB function for a 3x3 neighborhood is shown in Figure 26 
(MathWorks 2017c). The filter reduced noise levels within the image at limited expense 
to the runway edges and markings. An unfortunate consequence of median filtering is 
that the algorithm must first convert the original image to a grayscale image because it 
only filters pixel intensity values, thus adding additional computation time. Again, real¬ 
time application should only apply median filtering to the required areas of the image 

mask; this section only models the result of filtering due to constraints in MATLAB. 
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Figure 26. Intensity Comparison for Unfiltered and Filtered Runway Images 

D. IMAGE SHARPENING 

Image sharpening is yet another fundamental operation in image proeessing. This 
seetion provides a brief overview and background of the process, its purpose in the 
context of edge detection, and its formal application in the algorithm. 

1. Background, Description, and Purpose 

Image sharpening is a technique used to improve detail within an image, 
particularly textures and edges. Unsharp masking, a common sharpening technique, 
intentionally blurs the original image, usually through Gaussian blurring, and subtracts 
the blurred copy from the original image (Cambridge in Colour 2017). MATLAB 
sharpening functions allow the user to control the radius of the Gaussian fdter as well as 
the threshold for determining edges. A larger radius will affect a larger area around edges 
and higher thresholds will only apply the fdtering effect to stronger lines within the 
image (MathWorks 2017d). If the unsharp masking process identifies an edge that meets 
the threshold, the fdter sharpens the edge by an amount defined by the user. This 
“amounf ’ setting affects the acuteness of edges, which is a measure of edge transition 
from dark to light (Cambridge in Colour 2017). Higher acuteness leads to a sharper 
transition, hence the name of the operation. An application of unsharp masking is 
demonstrated in Figure 27, where an image was intentionally blurred with a Gaussian 
filter and sharpened using MATLAB’s “imsharpen” function from the image processing 
toolbox (MathWorks 2017d). 
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Gaussian blurred image (top left), sharpened image (top right), magnified view of blurred 

image edge (bottom left), and magnified view of sharpened image edge (bottom left). 

Figure 27. Demonstration of Unsharp Masking Using a Gaussian Blurred Image 

The unsharp masking process uses the intensity values of an image, but the output 
image can be in both color and grayscale. The function will simply convert the image 
between color spaces to meet the requirement. Figure 27 shows an ideal image 
sharpening process, with an edge between a perfect white background and a perfect black 
object. Real application of unsharp masking will almost never meet these ideal conditions 
but the process is still effective. For the runway detection algorithm, image sharpening is 
necessary because edge preservation is crucial for final application of the Hough 
transform. Without clear, defined edges, the edge detection process and resulting Hough 
transform calculation will not be accurate. The filtering process, while necessary, 
introduces additional blur to the image that sharpening can reduce. Although median 
filtering preserves edges to some extent, image sharpening refines all the major edges 
within the image and prepares the processed image for the final stages of runway 
recognition and detection. 

2. Application in Algorithm 

The objective of image sharpening in the context of the detection algorithm is to 
increase edge sharpness for lines on the runway while limiting the sharpening effect for 
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all other irrelevant areas. Variation of the radius, threshold, and amount inputs will 
achieve this objective. Ideal sharpening will only affect a small area around the edge of 
interest, so a smaller radius is ideal. The default radius of 1 pixel was used because it is 
an agreed upon standard and experimentation with lower and higher values yielded no 
improvement. Low thresholds will capture unnecessary details and textures within an 
image but excessively high thresholds will not effectively sharpen any edges. The 
thresholding and morphology steps executed prior to sharpening have eliminated most of 
the unnecessary regions in the image. For this reason, the algorithm uses a threshold limit 
of zero, accepting introduction of some noise as a result. The amount must be sufficient 
to highlight the edges within an image but not so high that the overshoot between the 
light and dark transitions of the line becomes unnatural or counterproductive to detection 
(Cambridge in Colour 2017). Through trial and error, the amount argument was set to 5, 
which significantly increased the sharpness of the runway edges without unnatural 
distortion in the image. The final product is shown in Figure 28. 
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IV. FRAMEWORK AND METHODOLOGY: LINE EXTRACTION 

AND ERROR CORRECTION 


This chapter describes the edge detection methods and line extraction functions 
specifically chosen to identify the runway edges within an image. It includes a complete 
analysis and comparison of relevant edge detection methods as well as a detailed 
overview of the Hough transformation process. 

A. EDGE DETECTION 

Edge detection is the final step before applying the Hough transform and 
calculating the position of the runway edges. This section provides a brief overview of 
edge detection, its purpose, and the various techniques used for edge detection. Also 
included are a comparison of methods and the justification for use of the Sobel operator. 

1. Overview and Purpose 

In a study and comparison of various edge detection techniques, Maini and 
Aggarwal (2009, 1) describe edge detection as “the process of identifying and locating 
sharp discontinuities in an image.” These discontinuities are the points at which the 
intensity of an image changes, indicating the start or end of an edge (Maini and Aggarwal 
2009, 1). The general approach to edge detection involves convolving a square operator 
with an image that detects these discontinuities and returns a binary image displaying all 
detected edges (Maini and Aggarwal 2009, 1). Gradient operators work by detecting the 
maximum and minimum values of the derivative of intensity values while Laplacian 
detection uses the second derivative to detect edges (Maini and Aggarwal 2009, 2). 
Laplacian methods are inherently more computationally intensive and adversely affect 
real-time detection, so this research ignores Laplacian methods in favor of gradient 
operators. 

2. Comparison of Edge Detection Techniques 

With the elimination of Laplacian methods, the two most popular gradient 
methods, the Sobel operator and the Canny algorithm, remain. The Sobel method 
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convolves a 3x3 gradient operator, shown in Figure 29, with the original image (Sobel 
2014). The gradient operator sums the gradient values of orthogonal vectors resulting in a 
magnitude and direction value for a given neighborhood (Sobel 2014). The user can 
therefore detect lines along a specific gradient, usually broadly categorized as horizontal 
or vertical, as well as lines of a certain magnitude. At the time of its inception, the Sobel 
operator was notable for its performance with other available edge detection techniques 
and its superior computation time. Developments in edge detection over the years have 
led to its new label as a traditional and simple edge detection technique, but the Sobel 
method still merits widespread attention and application in the image processing field. 
More involved than the Sobel method, the Canny method is actually a combination of a 
several processes. It vastly increases accuracy while maintaining an acceptable level of 
complexity. The algorithm achieves three objectives as outlined by Canny: low-error rate, 
accurate localization, and a single response to each edge (Maini and Aggarwal 2009, 6). 
The general process for the Canny method is as follows: application of a Gaussian filter, 
convolution with a gradient operator, calculation of gradient magnitude and direction, 
line thinning via non-maximum suppression, and hysteresis to eliminate any gaps in the 
edge (Maini and Aggarwal 2009, 6-8). 
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Figure 29. Vertical Sobel Operator (left) and Horizontal Sobel Operator (right) 

The Sobel and Canny methods share many of the same advantages and 
disadvantages. They are popular because they are simple to use and they can each capture 
edge direction and magnitudes. However, both are somewhat inaccurate and less robust 
than more complex edge detection techniques. When comparing the two methods, the 
Sobel operator is more computationally efficient, whereas the Canny method is more 
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accurate. Determination of the appropriate method depends on the trade-offs between 
advantages and disadvantages and the role and scope of edge deteetion in the greater 
system. The Sobel operator was the favorable method in this research for its simplicity 
and computational efficiency. Many of the steps taken in the Canny method were 
unnecessary due to previous image proeessing steps. For example, the Gaussian filter 
applied by the Canny method would be redundant and it would undo many of the 
favorable effects of the sharpening process. The difference between the two methods is 
best illustrated by their outputs, which are displayed in Figure 30. Each method was 
implemented in MATLAB with a vertieal filter and threshold set to 0.015. 



Figure 30. Comparison of Sobel Edge Deteetion (left) 
and Canny Edge Detection (right) 


3. Application in Algorithm 

The algorithm used Sobel edge detection via MATLAB’s “edge” function from 
the Image Proeessing toolbox. This function allows the user to specify the edge detection 
technique as well as a series of input arguments ineluding threshold, direetion, and 
thinning (MathWorks 2017e). The threshold for the Sobel operator was set to a 
magnitude of 0.015. Through trial and error, the author determined that this value 
performed best at all ranges. The direction was set to vertical, which executed 
eonvolution with the vertical Sobel operator. For final approach images, the runway 
direetion should align with the vertical axis of the frame. Resulting binary images from a 
variety of ranges are shown in Figure 31. 
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Figure 31. Sobel Edge Detection on Runway Images at 
Far, Medium, and Close Ranges, Respectively 

B. HOUGH TRANSFORM 

This section provides a brief background to the invention and development of the 
Hough transform, ultimately leading to its importance in the modem field of computer 
vision. Also included is an explanation of the transformation process and its application 
in the mnway detection algorithm. 

1. Background 

Paul Hough first introduced the concept of the Hough transform as a means to 
detect patterns created by subatomic particles in a bubble chamber (Hough 1962, 3). The 
motivation behind the Hough transform was to reduce the amount of time required to 
analyze photographs of the particle by allowing machines to analyze the pictures and find 
patterns (Hough 1962, 3). This motivation makes this process well suited for analyzing 
mnway approach images in real-time. Hough’s patent describes a plan to transform 
points in a Cartesian plane into the slope-intercept parameter space in which intersection 
points would correspond to lines in the Cartesian plane (Hough 1962, 3). If the objective 
of image analysis is line or pattern detection, the Hough space makes the process much 
less time intensive. Patterns of interest become points of intersection in the Hough space, 
reducing what would normally be a complex and time-intensive search of the Cartesian 
plane. 

Richard Duda and Peter Hart (1972, 2) improved the Hough transform by using 
polar coordinates to describe points in the parameter space via Equation 1.3. 

p = xcos,0 + ys,mO (4.1) 
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This eliminates the possibility of undefined or infinite slopes in the slope- 
intercept plane (Duda and Hart 1972, 1). The transformation to the polar parameter space 
is illustrated in Figure 32. Hart and Duda (1972, 4) were also able to establish the 
following rules for the transformation: all points in an image frame correspond to a 
sinusoid, points in the parameter space are straight lines in the image, and points on the 
same line in the image frame will share a common point in the parameter space. This 
improvement to the Hough transform makes it extremely applicable in image processing 
and computer vision. Undefined lines in the parameter space become a non-issue and 
image processing algorithms need only to find peaks in the Hough space to identify lines 
in the image plane. The more sinusoids intersecting at a single point, the stronger the 
peak, and the longer and more defined the edge or line. Most computer vision algorithms 
use Hart and Duda’s adaptation of the Hough transform. 



Figure 32. Transformation from x-y Image Plane to p-0 Parameter Space 

2. Application in Algorithm 

The Hough transform is the optimal process for identifying runway edges in an 
image. Considering its objective, the Hough transform is relatively efficient and it offers 
value in a real-time detection environment. The key to further reducing computation time 
is to limit the number of edges detected by the Sobel operator. Any clutter or excess lines 
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will only degrade accuracy and add to computation time. This is why the thresholding, 
morphology, and sharpening steps are so crucial to improving accuracy and efficiency. 

The Sobel edge detected image serves as the input to the Hough transform 
process. As with most processing steps, binary images with clear delineations between 
edges and limited noise provide accurate results. The “hough” command in MATLAB 
will perform the Hough transform, using the binary image as the input and producing a 
matrix that describes the Hough space with rho and theta values (MathWorks 2017f). For 
added accuracy and error reduction, the search of the image plane and the resulting 
Hough space is limited to theta values between -35 and 35 degrees. This means that the 
lines the Hough transform seeks to detect are vertical lines, with extra room to allow for 
variations in aircraft position relative to the runway in the final approach stage. The 
parameter space and the associated binary input image are shown in Figure 33. Each 
sinusoid corresponds to a pixel in the binary image and all possible lines passing through 
that image from -35 degrees to 35 degrees. 



Figure 33. Binary Input Image and Rho/Theta Hough Parameter Space 


A cursory glance at the Hough space will indicate a few likely intersection peaks 
corresponding to lines in the image plane. Points of specific interest are located near the 
negative 25-degree mark and the 25-degree mark in Figure 33. For the runway edges, two 
responses of similar magnitude but opposite angles are indications of an accurate match. 
If the aircraft is lined up with the center of the runway, the angles made by both edges of 
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the runway will be the same magnitude. In fact, these runway angles can aid in the 
estimation of aircraft roll positions. These peak intersection points are highlighted by 
yellow boxes in Figure 34 by using the “houghpeaks” command in MATLAB 
(MathWorks 2017g). This command is useful because it allows the algorithm to focus on 
a predetermined number of peaks. The two peaks with the largest number of intersections 
are most likely the runway edges, but this may not always be the case. For this reason, it 
is a good practice to analyze the top 5-7 peaks in the image, depending on the amount of 
noise and clutter. 
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Figure 34. Hough Space with Top Peaks Highlighted 

For the example shown in Figure 34, the top two peaks do correspond to the 
runway edges, but to prove this, the algorithm must convert the points back to lines in the 
image frame. To do this, the MATLAB function “houghlines” uses the peaks returned 
from the “houghpeaks” function to extract the appropriate rho and theta values that the 
function then into x-y plane values (MathWorks 2017h). Overlaying the resulting Hough 
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lines on the original runway image proves that the lines do accurately correspond to the 
runway edges, as shown in Figure 35. 



Figure 35. Runway Image with Hough-Detected Lines Overlaid 


Using a binary image collected via the Sobel method, the Hough transform can 
accurately detect the edges of a runway. The remaining challenge lies in correcting error 
and implementing the visual feedback in a control system. 

C. ERROR DETECTION AND ADJUSTMENT 

Error adjustment is critical to the success of the runway detection algorithm. If the 
algorithm is to be used in a real-time environment, it must achieve a high degree of 
accuracy. This section describes the process used to detect and correct errors within the 
algorithm. 

1. Detection Issues Using the Hough Transform 

The algorithm, as described in the previous sections, is robust; however, it will 
not provide total accuracy in all final approach scenarios. Attempting to pinpoint runway 
edges from a camera attached to a vibrating, high-speed vehicle is an inherently complex 
process. Issues are plentiful and experimentation with thousands of frames of final 
approach images exposed some of these problem areas. 
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As previously stated, the accuracy and speed of the Hough transform depends on 
the accuracy of the preceding steps of the algorithm. If thresholding and morphology can 
remove clutter and capture the entirety of the runway, then the Hough transformation 
becomes much simpler and more accurate. However, often the environmental conditions 
and image noise levels make detection much more difficult and simple operations such as 
thresholding, morphological opening, and closing are not suited nor were they designed 
to handle such complex scenarios. Therefore, noise and clutter in the input binary image 
are inevitable. Distance and environmental conditions that negatively affect visual range, 
such as clouds or fog, are common sources of these issues. 

Even in ideal conditions, the Hough transform can provide false outputs. 
Combinations of noise and pixel alignment may result in peaks in the Hough transform 
that do not correspond to any edge or runway line at all. This example is illustrated in 
Figure 36. Clutter and noise in the center of the runway caused the algorithm to fail to 
identify the right runway edge. For unavoidable situations like this, added error detection 
and correction is required. The challenge lies in reproducing human-like error corrections 
in an automated process. 



Figure 36. Inaccurate Runway Edge Detection via the Hough Transform Method 

2. False Runway Edge Detection Adjustment 

The algorithm tests for a series of conditions to ensure that the identified edges 
are correct. The first condition tests the proximity of the runway endpoints for both the 
right and left edges. Although the distance between these points will vary depending on 
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image resolution and distance from the runway, the algorithm defines a “no closer than” 
distance that indicates one or both of the edges are incorrectly identified. This “no closer 
than” distance is 20 pixels or less in the positive or negative x-direction. This distance 
represents half the distance between the runway edges in the image taken at the farthest 
point in the final approach of the aircraft. Therefore, there is no scenario in which the 
distance between runway edges should be any smaller. However, this value will also 
depend on camera properties such as resolution and stored image size so variation based 
on equipment and operating conditions is acceptable. 

The second error scenario tests for distances between the runway endpoints that 
are too large. The algorithm considers any distance between the far runway endpoints 
greater than 150 pixels in the x-direction or 40 pixels in the y-direction as too large. The 
largest distance between the far endpoints in any frame on the final approach data was 
100 pixels, so a value of 150 is provides a reasonable buffer. The far runway endpoints 
are the best choice for distance evaluation because there is less fluctuation in their 
separation compared to the closer runway endpoints. The separation between the runway 
start points can vary from 50 pixels at long distances to nearly 350 pixels near 
touchdown. The far runway endpoints simply provide a narrower range and reasonable 
values for error testing. 

The algorithm begins by assuming that the top two peaks in the Hough space 
represent the runway edges. If any of the previous error conditions are satisfied, the 
algorithm revisits the Hough space and analyzes the top six Hough peaks to find the 
correct runway lines. The system then compares the top Hough peak to the remaining 
four peaks and tests the runway endpoints for the same error scenarios. When the 
algorithm identifies two lines that meet the given conditions, these become the identified 
runway edges for the frame. If no matches are found, the algorithm simply chooses the 
top two peaks and accepts the error, assuming that later error correction steps will limit 
the impact. The system does not conduct a more extensive search of the Hough space 
because it would result in unacceptable increases in computation time. Unfortunately, this 
means that the algorithm might have to accept a certain level of error in the process. 
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An added layer of error correction comes in the form of averaging. As the 
algorithm detects the runway endpoints, it stores the values in a matrix that holds the 
previous 15 values. When enough points are stored in the matrix, the algorithm averages 
the current endpoint positions with the previous values. This limits the effect of any 
errors that made it through the previous endpoint-checking stage. It also smooths the data 
to limit small disturbances. It is important to remember that the camera collecting the 
data is attached to the wing of the aircraft, which is exposed to mechanical vibrations and 
turbulence from the environment. This inherent shaking is visible in sequential viewing 
of the approach frames. The motion of the aircraft results in noticeable changes in 
endpoint position. Smoothing the data will be essential in a control system environment. 
Averaging acts as a low pass-fdter, which eliminates small fluctuations that will 
negatively affect the performance of the aircraft control surfaces. If the control system 
receives constantly fluctuating position data from the runway detection algorithm, the 
control surfaces in the aircraft will also constantly fluctuate. Depending on the aircraft 
characteristics and flight dynamics, this fluctuation can reasonably result in a failure to 
land. Therefore, averaging frames is a good practice for future applications of the 
algorithm. Additionally, the camera captures images at 24 frames per second, so the 
algorithm is averaging the data in intervals shorter than one second. This should still 
provide a quick enough update rate to suit the control surfaces of the aircraft. 

The matrix used to store runway data also allows the system to compare 
corresponding endpoint positions between frames. In other words, if the algorithm 
identifies a runway edge in which any of the points significantly differs from that of the 
previous frame, the system will detect an error. Specifically, if any of the endpoints falls 
outside a radius of 10 pixels from the previous endpoint, the algorithm detects an error 
and replaces the current endpoint values with those of the previous frame. To ensure the 
system is not caught in a loop of ever-increasing errors, the matrix will reset if the system 
detects more than seven errors in a row. 

With all of these error-reducing measures combined, the accuracy of the system 
improves significantly. Depending on the nature of the operational conditions, the 
aircraft, and the camera, the details of the error detection conditions may vary. However, 
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the framework can be applied universally. The error correction capabilities of the 
algorithm are illustrated in Figure 37, where uncorrected and corrected versions of the 
same frame are shown side by side. 



Figure 37. Uncorrected (left) and Corrected (right) Versions of the Same Runway 

Approach Frame 
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V. RESULTS AND ANALYSIS 


This chapter reviews the performance of the runway detection algorithm based on 
accuracy and processing time. It will also provide analysis on which operations had the 
greatest effect on the overall performance of the algorithm. 

A. ACCURACY 

The algorithm processed 3000 images from a final approach in a Cessna 206 to 
Monterey Regional Airport. In total, this data covers approximately one minute and 
40 seconds of flight time. The first and final frames are shown in Figure 38 for visual 
reference. 



Figure 38. First (left) and Last (right) Images Used for Algorithm Evaluation 

This research defines accurate runway detection as identification within a 10-pixel 
radius of the true starting or ending points of the runway. For an image taken at medium 
range, a 10-pixel shift in the position of both the start and end points could change the 
estimated runway angle by three degrees and the estimated runway length by 10 percent. 
These are the maximum levels of acceptable variation that could still reasonable lead to 
safe landing conditions. Additionally, the runway edge must include no breaks or gaps 
along the length of the runway. 

In a test of all 3000 frames, the algorithm incorrectly identified the runway in 
114 frames, which corresponds to an accuracy of 96.2 percent. Four additional tests 
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ensured the accuracy of the results. This test also utilized the error correction techniques 
of the algorithm. The accuracy of the detection algorithm without the error correction 
techniques was 75.9 percent. 722 frames of the original 3000 frames did not meet the 
accuracy standards, most of which were frames near the end of the final approach. This 
indicates the relative importance of the error correction process. It also indicates the 
negative effect that low altitude has on the detection algorithm. At lower altitudes, the 
runway consumes more of the image and its shape is comparatively more distorted than 
those at higher altitudes. 

Frames that led to inaccurate detection of the runway had a few commonalities, 
the first of which is being taken from either extremely long or extremely short distances. 
The majority of the errors came from frames at the beginning or end of the runway data, 
which means that long or short distances and high or low altitudes degrade the 
performance of the image processing algorithm. The altitude ranges along the flight path 
at which accuracy decreases is shown in Figure 39. 




Image 

Figure 39. Accuracy Error Related to Altitude and Heading 

At long distances, the most likely source of error is noise and environmental 
interference. Even in near-perfect visibility conditions, the clearness of the image will 
only improve as the distance becomes shorter. For the algorithm, this means that edges 
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are less delineated and sections of the runway can blend in with the surrounding tarmac 
of the airport. At shorter distances, the aircraft is closer to the ground and the view of the 
runway is comparatively distorted. Generally, these frames only include the concrete and 
asphalt surfaces of the runway and the horizon line. Saturation and value characteristics 
become increasingly more important at these distances as a result. Losing differentiation 
in hue will undoubtedly result in a loss of accuracy. It is also apparent that the runway 
endpoints approach the vanishing point in the image at shorter ranges. As a result, the 
endpoints become even more difficult to identify. The image frames that correspond to 
the start of processing degradation are shown in Figure 40. 



Figure 40. Maximum (left) and Minimum (right) 

Distance for Image Processing Degradation 


Clutter was also a common factor in incorrectly identified frames. Often larger 
and brighter elements near the runway, such as buildings, would pass through the 
filtering and morphology steps. If these elements happened to align with one of the 
runway edges and they were close enough to the runway surface, the algorithm often 
identified the clutter as continuations of the runway edge. Normally, the error correction 
process will identify these errors, but sometimes there are no alternative peaks in the 
Hough space to replace the false peaks. 

There is room for improvement in accuracy, and the level of accuracy will depend 
on the application of the algorithm. If the system is intended for use on smaller, easily 
controllable aircrafts and large runways, accuracy is not as much of an issue. Application 
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in an aircraft carrier environment would require nearly perfect accuracy. These are 
extensions of the system in which the operator may need to make adjustments to achieve 
increased accuracy, but this research proved that the basic framework can operate at a 
successful level. 

B. PROCESSING TIME 

MATLAB scripts and functions are not always conducive for evaluation of real¬ 
time application. An aircraft would never use a MATLAB environment to implement 
real-time computer vision and aircraft control. MATLAB functions often call upon 
various toolboxes and sub-functions that require additional loading times. Sometimes 
MATLAB loads and implements subroutines in C or C++ to execute a function, as is the 
case for the filtering operations. However, it is much easier to demonstrate the framework 
and methodology in MATLAB. MATLAB can still offer insight and broad implications 
for real-time evaluation. Therefore, this discussion will only offer recommendations for 
reducing processing time based on MATLAB execution. 

Data collected from five approach frames processed five times each revealed 
patterns as to which functions and operations consume the most processing time. The 
images vary in their distance from the runway to avoid bias due to distance. The “imfiH” 
function, median filtering, and image sharpening were consistently the top three time- 
consuming processes. These are all matrix operations or convolution processes that 
must calculate values for every pixel in the image so it is not surprising that they are the 
most computationally inefficient. On average, the filling, sharpening, and filtering 
operations consume 11.2 percent, 10.5 percent, and 10.2 percent of the total processing 
time, respectively. 

A possible explanation for the computation time of the “imfill” function is that it 
is not a true closing operation. It is a hybrid function that uses a more complex technique 
to fill in holes in binary regions of the image (MathWorks 2017a). The algorithm also 
uses two iterations of the “imfill” function, further explaining its increased processing 
time. While the computation is time consuming, it is still worthwhile to include both 
iterations of the morphological operation. 
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Image filtering is diffieult to avoid, but limiting the area that requires filtering is a 
good practice for cutting computation time. The equipment used is also vital for the 
determination of the level of noise the algorithm must handle. Some cameras can provide 
clearer images with less noise, but they will come at a cost. Strategic placement of the 
filtering process in the algorithm will ultimately determine its efficiency. If implemented 
after all morphological operations and masking, filtering will be much more 
computationally efficient. 

Image sharpening is the only process that could reasonably be removed from the 
algorithm, but only if conditions allow. Again, this is largely a factor of the operating 
environment, specifically the visual conditions, and the level of noise due to camera 
operations. With these factors reduced below an acceptable level, the algorithm can 
reasonably provide accurate results without the sharpening process. However, achieving 
those conditions is not easy, and it will likely be expensive. Efficient use of the 
sharpening threshold and amount settings can further reduce the computation time if 
sharpening is still necessary. 
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VI. CONCLUSIONS 


This chapter offers a comprehensive review of the researeh objective and foeus, 
the framework and methodology of the runway detection algorithm, and results. It also 
offers lessons learned from the proeess and options for future work. 

A. REVIEW OE RESEARCH QUESTION 

The objective of this researeh was to eonduet a feasibility study for the 
application of a runway detection algorithm for use in a vision-based autonomous landing 
system via a systems engineering approaeh. The system should be evaluated on its ability 
to deteet the edges of a runway and any other relevant runway markings in an aeeurate, 
robust, and eost effieient manner. On a broader seale, this researeh assesses the feasibility 
and applieation of vision-based eontrol in a broad array of autonomous systems. 
However, unmanned aerial vehicles propose a unique ehallenge to developing robust and 
autonomous decision-making systems. 

B. CONCLUSIONS 

Computer vision ean accurately deteet and traek a runway in the final approach 
stages of flight. This researeh outlined a framework and methodology that proved its 
worth with an aecuracy of 96.2 percent in final approaeh testing. Additionally, the 
framework supports real-time implementation in an aireraft control system. 

Applying computer vision techniques to runway detection and unmanned vehicles 
requires a system that is aeeurate, robust, and computationally efficient. An aeeurate 
algorithm will utilize a multi-stage approach that constantly works to localize the runway 
region and eliminate noise and elutter. The algorithm should also utilize error detection 
and correction techniques to avoid false positives or misidentifieation of runway edges. 
These layers of image proeessing steps add an aspeet of redundaney to the algorithm and 
improve overall performanee. 

A robust algorithm will be able to work at all distances expeeted in the operating 
environment, through various levels of noise, and in a degraded visual environment. It is 
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easy to construct an algorithm that will detect a runway in a very limited set of 
circumstances. The challenge lies in designing a system unaffected by external factors. 
Hue, Saturation, and value thresholding is a convenient way to increase robustness 
because it allows the algorithm to use more than just the intensity characteristic to 
localize the runway at limited computational expense. The more data available to the 
system, the more accurate it can be. Median fdtering is also an efficient way to reduce 
Gaussian, salt and pepper, and quantization noise that will undoubtedly be present in an 
image. Combining these strategies will increase the operating range of the system. 

Lastly, a computationally efficient algorithm will accurately detect the runway 
edges in the minimum amount of time, ideally for implementation in real-time. If the 
algorithm limits the amount of noise and clutter in an image, it will reduce the 
computation time for operations that require convolution and other complicated 
procedures like the Hough transform. It is acceptable to sacrifice accuracy in favor of 
computation time if the results are still reasonably in the desired range, as is the case with 
using the Sobel operator instead of the more accurate Canny method. This framework 
strikes a delicate balance between efficiency and accuracy. 

C. FUTURE WORK 

Future work would primarily include implementation of pose and attitude 
estimation as well as the development of a full aircraft control system with visual 
feedback as outlined in Chapter I. By doing so, the feasibility study could be extended to 
the complete vision-based landing system. 

While this thesis conducted a thorough description and analysis of the image 
processing system within the constraints of the available data, the system still requires 
additional testing. Specifically, future work should collect data from a wide variety of 
runways and geographic locations to test the effects of background and clutter on the 
accuracy of the algorithm. Additional testing should also cover various light levels, 
degraded environmental conditions, different types of aircraft, varying speed, and varying 
glide slopes. If the landing system is to be truly robust, it must successfully operate in all 
of these conditions. 
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While the results of the thesis have proven that a vision-based landing system is at 
least partially feasible, they also indicate that the runway detection algorithm struggles at 
extremely long and short ranges. If this vision-based approach is to be pursued, future 
work must focus on achieving accurate results at both long and short ranges. Perhaps the 
solution to this issue lies in adapting the algorithm itself. The system could also possibly 
use integrated sensors such as altimeters or range finders to increase accuracy. 
Regardless, further development and analysis is required to increase the range of the 
system. 
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APPENDIX. HSV AND GRAYSCALE PROCESSING TIMES 


Trial 

Grayscale Time (s) 


1 

0.168 

0.072 

2 

0.151 

0.043 

3 

0.158 

0.040 

4 

0.195 

0.043 

5 

0.195 

0.046 

6 

0.185 

0.050 

7 

0.190 

0.058 

8 

0.194 

0.047 

9 

0.265 

0.053 

10 

0.192 

0.052 

11 

0.195 

0.048 

12 

0.231 

0.055 

13 

0.192 

0.052 

14 

0.294 

0.047 

15 

0.187 

0.048 

16 

0.178 

0.047 

17 

0.194 

0.048 

18 

0.227 

0.054 

19 

0.184 

0.057 

20 

0.193 

0.047 

21 

0.197 

0.047 

22 

0.189 

0.054 

23 

0.210 

0.073 
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